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ABSTRACT 



This evaluation report of the Family to Family Program 
assesses parental attitudes towards their Family to Family experience and the 
functioning of their emotionally impaired children. It reviews issues of goal 
achievement; the impact on the targeted problem; service population 
demographics; and sustainability. Related topics include administration and 
administrative support; recruitment of consumer families; identification of 
host families; consumer support services; and characteristics of the 
consumers and their families. The report assesses all four variables proposed 
in the evaluation plan including: the functioning of participating children, 
family cohesion and adaptability, and parental attitudes about personal and 
family issues using the Family to Family Questionnaire. The evaluation 
concluded that the program succeeded .in achieving its stated goals of 
providing appropriate families with respite services. Child, family, and 
parent variables measured by the three instruments all yielded significant 
evidence for the efficacy of the program. All statistical indices suggest 
that children, families, and parents function better as a result of the 
program's services. (JDM) 
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Introduction and Scope of Report 



This evaluation report covers the period from March, 1999 through April, 15, 2000. It 
will cover issues of goal achievement, impact on the targeted problem, service 
population demographics and sustainability ( Skillman Foundation, 1996) . Related 
topics include administration and administrative support, recruitment of consumer 
families, identification of host families, consumer support services and characteristics of 
the consumers and their families. The children served by the program will be described 
based on clinicians’ ratings on the Child and Adolescent Functional Assessment Scale 
(CAFAS) (Hodges, 1990). Parental attitudes toward the program will be described 
using the Preplacement Questionnaire and family dynamics by the Family Cohesion 
and Adaptability Scale II (FACES II) (Olson, et al. 1986). This report will therefore 
assess all four of the variables proposed in the Evaluation Plan. These variables and 
the means of assessment are: 

1) Functioning of participating children. (CAFAS) 

2) Family cohesion. (FACES II) 

3) Family adaptability. (FACES E) 

4) Parental attitudes about personal and family issues which the program is designed 
to ameliorate. (Family to Family Questionnaire) 

As noted in previous reports, these data were gathered on all enrolled families and 
children at the time of enrollment. The Evaluation Plan calls for the same instruments 
to be administered post treatment as well. A minimum treatment of six months has 
been agreed upon for all children in the program for evaluation purposes. Pre and post 
data were to have been collected when cases of at least six months program duration 
were closed and for all cases of six months duration or more at the end of the third 
year of program operation. Unfortunately, there have been difficulties in gathering the 
post data due to a number of causes, including family relocation and remissness of 
clinicians in supplying CAFAS data. Moreover there was uncertainty on the part of the 
program staff as to whether a third and final Evaluation Report was required. The 



evaluators were told in February that no report was required; later that one would 
indeed be needed, and the May deadline was negotiated. 

Accomplishment of Program Objectives 

The overall goal of the Family To Family demonstration project was to keep families of 
emotionally disturbed children intact by supplying volunteer respite providers (who 
are paid a small stipend) within the community. The proposal considered that the 
program might alleviate several aspects of the problems faced by the families of special 
needs children who are emotionally disturbed. These can be considered mediate 
program objectives: 

1) Need for respite. Parents of these children experience serious, even 
overwhelming stress in carrying out the relentless demands of parenting. Respite care 
has been shown to provide an important and restorative break in these demands, and 
provide the entire family with benefits to their well being. 

2) Family isolation. Extended family and fiiends may avoid a family with 
emotionally disturbed children, leaving them isolated. The intent of the program is to 
provide these families with opportunities to interact with family and fiiends during 
respite periods and in some cases to involve extended family or fiiends in providing 
respite. 

3) Crises. Theses families are at serious risk for crisis situations which they may be 
ill equipped to manage. Respite care can reduce the incidence and severity of crises by 
reducing die feelings of anxiety, being overwhelmed, and chaos which are endemic in 
families with emotionally disturbed children. 

4) Lack of parental role models. The program seeks to provide children with 
additional parental role models through respite care and to improve parenting skills in 
file family of origin through various supportive services (Family to Family Proposal, 
1996). 

Specifically, the Family to Family Program has sought to provide respite care for 25 
families with emotionally impaired children through regular out of home care with host 
families. As noted, this is intended to keep consumer families intact, and to improve 
family functioning and quality of life, hi order to accomplish these ends, the Family to 
Family program engaged in three major activities: 

1) Identifying and enrolling appropriate consumer families; 

2) Recruiting suitable host families and arranging the respite care; 

3) Providing ongoing support to both consumer and host families. 

This report will evaluate the program in terms of those activities over the past year, as 
well as the level of administrative support on the part of the Oakland County 
Community Mental Health Authority. In addition, we will examine the impact on the 
consumers and their families through analysis of pre and post data gathered by the 
Family To Family staff. The evaluative process therefore attempts to respond on the 
macro level of meeting a community need, on the mezzo level of improving the quality 
of life of particular family constellations, and at the micro level of targeting changes in 
the behavior of individual emotionally disturbed children. 
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Identifying and Enrolling Appropriate Families 

The Family To Family Program has depended upon OCCMHA clinicians to identify 
from their caseloads appropriate children and families for referral to the program. 

After some difficulty with start up, and a period of underenrollment relative to 
program capacity, the program currently has 23 children from 21 families (April, 15, 
2000). Because a number of enrolled families have moved from the area, or 
terminated their participation for other reasons, recruitment has been an ongoing 
activity for the staff, requiring constant effort and communication with the clinicians to 
obtain referrals. That these efforts have paid off is evident from the near full 
enrollment, and occasionally a waiting list. . Reasons for termination are: moved from 
area / unable to contact (1 1), unsuitable for the program (5), placement in foster care 
or restrictive environment (3), and surrender of parental rights (1). The program staff 
report a good level of cooperation from clinicians and an increasing collaboration from 
those in the previously underrepresented northern area. Overall, the recruitment of 
suitable consumers is evaluated as satisfactory. 

Recruiting Suitable Host Families 

The recruitment of suitable host families for die emotionally disturbed children 
enrolled in the Family To Family Program has presented some problems, due to the 
nature of the special needs children. However with considerable ingenuity and utilizing 
the extended families and friends of the children’s families of origin, every case has 
resulted in at least some respite care, and in nearly all cases a mutually satisfactory 
arrangement of respite has been worked out. The stipend for host families which was 
$45 per day has been increased to $65, a needed improvement. The maximum number 
of respite days is 108 per year (9 days per month), so that some creativity is required to 
reach a satisfactory accommodation. Considering the difficult children who have a 
high incidence of hyperactive, impulsive and aggressive disorders enrolled in this 
program, we evaluate this aspect of the program to be satisfactory. 

Providing Ongoing Support 

It is apparent to the evaluators from their contacts with program staff that the Oakland 
County Community Mental Health Authority has provided appropriate training, 
supervisory and physical support for the program. The original Family to Family 
Program Coordinator, Christine Miller, left the position and has been replaced by Terry 
Clisshold who has carried out the job in a diligent and professional manner. Her 
supervisor, Michelle Quarton, who oriented and trained Ms. Clisshold, has also been 
replaced by Kris Mazzei. The Program Coordinator is responsible for the day to day 
operation of the program, carrying out numerous activities to encourage clinicians to 
refer appropriate consumers to the program, to locate and negotiate with host families, 
to assist enrolled families with various problems, to provide informational and 
educational support for consumer and host families, and to carry out administrative and 
recording activities. The Program Coordinator, Ms. Clisshold, has provided these 
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services to enrolled families in a timely and supportive manner. She has produced a 
quarterly newsletter for the program participants and conducted various meetings and 
training sessions for families as needed. We consider her commitment, 
professionalism and interpersonal skills to be indispensible elements in the success of 
the program this past year. As for the other aspects of providing for the Family to 
Family program: technical, secretarial, space, supplies and other material resources; all 
seem adequate to the needs of the enterprise. Consequently, the provision of support to 
various constituents is judged to be satisfactory. 

Consumer Demographics 

Consumers have ranged in age from 5 to 17 with a mean for those currently enrolled 
of 12 yr., 8 mos. The 13 consumers enrolled during the past year ranged from 9yrs. 0 
mos. to 17yrs. 0 mos. with a mean age of 12 yrs. 10 mos. Of these 9 were males and 
4 were females. The current enrollment is divided into 18 males and 5 females. Two 
families, it should be noted, have two children in the program. 

CAFAS Measurements 

The CAFAS yields 4 scores (0 = no/minimal impairment; 10 = mild impairment; 20 = 
moderate impairment and 30 = severe impairment) on five scales listed below. 

1. Role performance - subscales are schooFwork, home and community. 

2. Behavior toward others. 

3. Moods/self harm - subscales are moods/emotions and self harmful behavior 

4. Substance abuse 

5. Thinking 

Where there are subscales, the highest of the scores is used. The 5 Scales are summed 
to yield a Total Youth Score ranging from 0 to 150. Clinical indicators for Total Youth 
Scores are: 

10 = may benefit from some intervention or prevention. 

20-30 = treatment on an outpatient basis assuming no risk factors (suicide, self or 
other harmful due to aggression, sexual behavior or firesetting, or psychotic/organic 
symptoms with severe incapacitation. 

40-60 = may need services beyond weekly outpatient visits, but not likely residential 
treatment. 

70-80 = may need intensive therapeutic program, depending on resources and risk 
factors. 

90 or higher = restrictive/supervised living situation may be needed. 

The Family to Family Program appears most suitable for children of moderate CAFAS 
Total Youth Scores, though those with more severe dysfunction might also benefit with 
additional supportive services. In fact we do find a wide range of Total Youth Scores: 
from 30 to 1 10 at entry. The mean entry Total Youth Score of current consumers in 
the program was 64.3, indicating a midrange level of impairment and a need for 
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services beyond weekly outpatient care. This is comparable to the scores reported in 
previous reports: 66.5 last year, and 64.5 two years ago. Of the 23 currently enrolled 
program consumers, 4 had entry CAFAS Totals of 90 or more, indicating a need for 
careful monitoring and high level supportive services. Case notes indicate that all these 
cases are working out with the host families. Mean entry CAFAS scores of current 
consumers on the five scales along with numbers of consumers falling into the three 
impairment levels are summarized in Table 1. 



Scales Impairment Levels 





Mean Scores 


# Severe 


# Moderate 


# Minimal 


Role Performance 


22.7 


10 


9 


4 


Behavior toward Others 


13.0 


1 


9 


13 


Moods/ Self Harm 


18.6 (Moods = 


14.3) 3 


10 


10 


Substance Abuse 


0 


0 


0 


0 


Thinking 


4.78 


1 


3 


19 



Table 1. CAFAS Scores, Current Enrollees (n = 23) 

Of the 23 current cases, 7 had scores of 70 or higher, 12 had scores of 40 to 60, and 4 
had scores of 30 or less. As table 1 indicates, Role Performance was the most 
common severe impairment and had the highest mean score; Moods was next in mean 
score and had the largest number of moderate impairments; Behavior toward others 
was also a common problem. Self Harm, and Thinking were less common and less 
severe, while Substance Abuse was not a problem in this group. 

Risk Factors were evaluated in 7 of the cases; of these, 5 were for aggression, and 2 for 
other factors. The number of aggressive risks is supported by DSM IV diagnosis of 
oppositional defiant disorder in 6 cases and attention-deficit/ hyperactivity disorder in 
13 cases (some with ODD). A diagnosis of dysthymia Occurred in 3 cases. No other 
DSM IV diagnosis appeared more than once. 

The two optional scales, Caregiver Resources: Material Needs, and Family/ Social 
Support were rated in 20 cases. Table 2 summarizes these ratings of the primary 
caregivers. 

Caregiver Support: 



Impairment levels 


Material 


Family/Social 


0 - None 


9 


1 


10 -Mild 


9 


9 


20 - Moderate 


2 


9 


30 - Severe 


0 


1 


Mean Caregiver Support Level: 


7.5 


15 



Table 2: CAFAS Caregiver Support Ratings (n=20) 
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It is clear from Table 2, that this group of families does better at providing for their 
children’s physical needs than their social and emotional needs, and providing a home 
setting free from such risks as abuse or parental alcoholism. It is noteworthy that just 
one family was rated as having severe impairment in either category. The mean rating 
indicates problems in the social support area from mild to moderate, so that staff 
monitoring of the consumer family situation was in order. These support ratings are 
similar to last year’s, when means for material and social support were 7.2 and 14.4. 

Statistical Analysis of CAFAS Scores 

CAFAS scores were gathered prior to entry into the Family to Family Program and at 
exit, providing the six months requirement was met, or at the end of the three year 
cycle, set at April, 15, 2000. These pre and post scores were then analyzed, employing 
the t- test for nonindependent samples. This technique is also referred to as the 
correlated, or dependent t-test The measure which is analyzed by this test is the mean 
difference between the paired scores. Table 3 shows the relevant details of the 
analysis. 

Variables N Mean St. Dev. df t Sig. (1 tailed) 



Pre CAFAS 25 64.4 25.51 

Post CAFAS 25 48.4 22.85 24 3.024 .003 



Table 3: CAFAS Statistical Summary 

Table 3 reveals a t-test value of over 3, in this case, significant at <.003 Thus we 
would predict this strength of relationship to occur by chance in fewer than three in ten 
thousand cases. This far exceeds the .01 level of confidence usually' specified in social 
research. The one tailed test was called for because there was reason to believe that 
CAFAS scores would decline (i.e. improve) with the application of respite and other 
services offered through the OCCMHA. A finding of this magnitude supports the 
clinical hypothesis that the Family to Family Program contributes to the well being of 
the the children whose families are in the program. 

Measures of Family Cohesion and Adaptability: the FACES II 

The Family Adaptability and Cohesion Scales have been developed over the past two 
decades by D. H. Olson and colleagues of the Family Inventories Project at the 
University of Minnesota. These instruments are based on the Circumplex Model of 
family functioning (Olson, 1983); and were described in last year’s report. However to 
spare the reader a search for that document, we offer this brief explanation of the 
FACES II instrument and its underlying rationale. The FACES II instrument measures 
two dimensions of family functioning: 

- cohesion: the degree to which members are connected to / separated from the family. 

- adaptability: the extent to which the family is able to change and be flexible. 
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FACES II linear scoring results in 4 levels of cohesion ( disengaged, separated, 
connected and very connected), and 4 levels of adaptability ( rigid, structured, flexible 
and balanced). There are 16 possible combinations of these 4 levels with each other 
yielding 4 family types. Figure 1 illustrates these types and the threefold clinical labels 
associated with them— balanced, midrange and extreme. 



FACES li: Linear Scoring & interpretation 
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FIGURE 1.! FACES It: Linear Scoring & Interpretation 



As noted in last year’s report, the FACES II instrument has good reliability (Cronbach 
alpha’s of .87 for the Cohesion scale, .78 for the Adaptability scale, and .90 for the 
total scale). Validity of the instrument is supported by concurrent studies and a well 
articulated and persuasive underlying theory— the Circumplex model of family 
functioning (Olson et al., 1983, 1991). 

For the purposes of this study comparisons were made between pre and post Cohesion 
scores, pre and post Adaptability scores and pre and post Family Type scores (1 
through 8). Pre and post FACES II measures were available for 26 families of six 
months or longer duration or longer in the program. As before the statistical method 
used to analyze these data was the the t-test for nonindependent samples. A directional 
(one tailed) test of significance stems from the initial hypothesis that respite care would 
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result in improved family functioning (OCCMHS Family to Family Proposal, 1996). 
Table 4 shows the results of the statistical analysis performed on pre versus post 
FACES II scores. Higher scores on all these variables indicate better functioning. 



Variables 
N Mean 


St. Dev. 


df 


t 


Sig. (1 tailed) 




Pre Cohesion 


26 


50.54 


12.60 






Post Cohesion 


26 


56.62 


8.66 


25 -5.043 


.00003 


Pre Adaptability 


26 


40.31 


7.48 






Post Adaptability 


26 


42.73 


6.01 


25 -.4.239 


.00003 


Pre Type 


26 


3.40 


1.57 






Post Type 


26 


3.92 


1.55 


25 -3.863 


.0005 



Table 4: FACES II Statistical Summary 



The results from the analysis above clearly show improvement in all three aspects of 
family functioning as measured by the FACES II test. Both Cohesion and 
Adaptability scores were literally off the scale in terms of significance. The authors had 
to consult a number of charts before estimating the significance levels due to these 
extreme t-test values The SPSS program used to run these data only supplies such 
levels to three decimal places (two tailed). This is a problem most researchers would 
be delighted to face. The odds against such a difference occurring by chance are three 
out of a million for Cohesion and Adaptability scores and five in one hundred 
thousand for the Family Type scores. These extremely significant values strongly 
support the hypothesis that the combination of services, including respite care through 
the Family to Family program, improves family functioning. As we shall note in the 
discussion later, we cannot isolate with certainty the exact role that the program played 
in bringing about these results, due to the nature of the research design. Nevertheless 
the change did clearly occur in the hypothesized direction. 

Family to Family Questionnaire 

This questionnaire was developed from an earlier longer instrument developed by the 
evaluators for the Association of Retarded Citizens of Oakland County program of 
respite services (Ramey and Meyer, 1990). It consists of 10 questions relating to the 
goals of the Family to Family program; the items are arranged in a 4 point Likert type 
scale with 1 being a desireable lack of problems/high functioning, and 4 representing the 
presence of problems/low functioning. Parent(s) are asked to rate the following areas: 

- ability to maintain the child(ren) at home (Ql) 

- stress level for self (Q2), and family (Q3) 

- time available for self (Q4), other children (Q5), spouse (Q6), and friends (Q7) 

- ease of childcare arrangements (Q8) 

- sense of control over one’s life (Q9) 

- adequacy of outside support (Q10) 
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The Family to Family Questionnaire is easy to administer, score and interpret and 
derives construct validity from its direct connection to the goals of the program. Pre 
and post Family to Family Questionaires were availaible for 27 parents of children in 
the program and the overall results were submitted to statistical analysis using the t-test 
for nonindependent samples. Table 5 shows the results of this analysis. 

Variables N Mean St. Dev. df t Sig.(l tailed) 



Pre Questionaire 27 3.10 .593 

Post Questionaire 27 2.19 .530 26 8.044 .000001 



Table 5: Family to Family Questionnaire Statistical Summary 

The t value of over 8 is significant at a better than one in ten million level of 
confidence. Little more can be said-- this is not a chance relationship. It is clear that 
parents improved significantly overall in measures covered by this instrument. The 
largest gain scores were for Q7, time available for friends (-1.44); Q8, ease of childcare 
arrangements (-1.37); Q10, adequacy of outside support (-1.27); and Q4, time 
available for self (-1.23). All question scores improved, with a mean improvement 
score of 0.91. This is a phenomenal improvement on a 4 point scale, as the t-test 
results reveals. Table 6 shows the means and gain scores fo each of the to Family to 
Family Questionnaire items. 



Question 


1 


2 


3 


4 


5 


6 


7 


8 


9 


10 


Pre Mean 


2.35 


2.91 


2.91 


3.56 


3.12 


3.21 


3.88 


3.37 


2.83 


3.15 


Post Mean 


1.74 


2.15 


1.96 


2.33 


2.33 


2.58 


2.44 


2.00 


1.89 


1.88 


Gain Score 


0.61 


0.76 


0.95 


1.23 


0.79 


0.63 


1.44 


1.37 


0.94 


1.27 



Table 6: Family to Family Questionnaire Item Analysis 



Inspection of Table 6 reveals that Ql, ability to maintain child at home and Q6, time 
for spouse, had the lowest gain scores, quite a bit lower than all others. We may 
speculate that respondents were either loathe to admit problems in these two crucial 
family areas of responsibility or that they placed more emphasis on these two aspects of 
family life. But even these items improved by a relatively large amount, given the 4 
point scale. This table shows the magnitude and pervasiveness of gains in important 
measures of family well being. Moreover it should be noted that items with pre means 
of 3 or more, indicating more severe problems, tended toward the largest gain scores— 
most over 1 point.. These problem areas, in order of severity were: Q7, time for 
friends; Q4, time for self; Q8, ease of childcare arrangements; Q6, time available for 
spouse; Q10, adequacy of outside support; and Q 5, time available for other children. 

In sum, we consider the difference in pre and post Family to Family 
Questionnaire scores to be strong evidence for the success of the program in meeting 
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consumers ’needs and enhancing the quality of their lives. The fact that the instrument 
was designed specifically to tap into the program’s objectives is a strength, but as a 
result is nonstandardized, of unknown reliability and only construct validity. 

Methodological Issues 

As in all program evaluation ventures, compromises and tradeoffs inevitably detract 
from the ideal design. This undertaking was no exception. Ideally we would have 
chosen a true experimental design— a post-test only control group design, for example. 
However, reality intervened and we were unable for obvious administrative and ethical 
considerations to realistically plan such an experimental approach. No large pool of 
prospective consumers existed: clients were referred by clinicians from their caseloads, 
and it was difficult to ramp the program up to speed; only a limited number of cases 
were deemed suitable for the program. Even if more potential consumers had been 
available, the ethical dilemma of denying service to suitable families was 
insurmountable. Therefore we proposed a quasi experimental evaluation employing 
pre and post measures on a single group of consumer families. Even so, it was not 
easy to track down families and obtain the post data. It was due to the tenacity of the 
Family to Family Coordinator, Terry Clisshold, that an adequate number was obtained, 
permitting statistical analysis. However, the quasi experimental design has weaknesses. 
These weaknesses include rival hypotheses accounting for changes that would in a 
stronger design be confidently assigned to research hypotheses. The plausible rival 
hypotheses include history, maturation,testing effects, and statistical regression (Isaac 
and Michael, 1977). Let us consider each of these briefly with respect to this 
program. 

- History: Something may have occurred during the interval between pre and post 
testing which might influence the outcome, hi this case children in the program did 
continue to receive clinician services and possibly other non program attention (e.g. 
medication). 

-Maturation: This is especially important with young children. As they develop, 
rather considerable behavioral and psycho/physiological changes occur, which may in 
some measure account for the observed changes. 

-Testing effects: the mere act of pretesting can create differences in the posttesting, 
or sensitize program participants to certain issues which modify their posttest 
performance. 

-Statistical regression: This occurs when an extreme group is used in the study, 
which very well may have been the case with Family to Family participants. The 
extreme scores, when repeated, will regress toward the mean, accounting for some of 
the change. 

It is likely, that some or all of these factors were at work in this evaluation study. They 
should be bom in mind when interpreting what otherwise seems to be a very 
convincing statistical case for the efficacy of the Family to Family program. 
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Conclusions and Discussion 



This evaluation concludes that the Family to Family program has succeeded in 
achieving its stated goals of providing appropriate families with respite services in order 
to enable the families to maintain the children) at home, to enhance family quality of 
life, to reduce family crises and to provide parental role models for the children. The 
program staff and clinicians have collaborated to maintain enrollments at or near the 
intended capacity of 25. The Program Coordinator has also managed to assist families 
to locate suitable host families to provide respite care in all cases without undue delay. 
We also found considerable evidence of appropriate case intervention where necessary, 
and ongoing contact by the Program Coordinator with the enrolled families. 

The transition from Skillman Foundation supported demonstration project to regular 
OCCMHA program has gone smoothly with existing personnel remaining on staff. 
Funding for the program has shifted to the general ledger of OCCMHA . 

Child, family and parent variables were measured by means of three instruments: the 
CAFAS, the FACES II and the Family to Family Questionnaire, all administered pre 
and post at least six months treatment. Ratings of children by clinicians on the CAFAS 
improved; t-test analysis revealed a .003 level of significance. FACES II results were 
also significant at the .00003 level for cohesion and adaptability, and at the .0005 level 
for type. The analysis of the Family to Family Questionnaire data also yielded an 
astronomical significance level: .000001. While this is powerful evidence for the 
efficacy of the program, we urge cautious interpretation due to the quasi experimental , 
pre- post test design . 

Overall, we must conclude that the Family to Family program has been a resounding 
success. All clinical and administrative oversight indicates attainment of all program 
goals and objectives, a competent and committed staff, appropriate clients, suitable 
services and smooth integration into OCCMHA’ s regular program offerings. Beyond 
that, all statistical indices suggest that children, families and parents function better as a 
result of the program’s services. 
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